Providing track format information when mirroring updated tracks from a primary storage system to a secondary storage system

ABSTRACT

Provided are a computer program product, system, and method for providing track format information when mirroring updated tracks from a primary storage system to a secondary storage system. The primary storage system determines a track to mirror to the secondary storage system and determines whether there is track format information for the track to mirror. The track format information indicates a format and layout of data in the track, indicated in track metadata for the track. The primary storage system sends the track format information to the secondary storage system, in response to determining there is the track format information and mirrors the track to mirror to the secondary storage system. The secondary storage system uses the track format information for the track in the secondary cache when processing a read or write request to the mirrored track.

BACKGROUND OF THE INVENTION 1. Field of the Invention

Provided are a computer program product, system, and method providingtrack format information when mirroring updated tracks from a primarystorage system to a secondary storage system.

2. Description of the Related Art

In a storage environment, a host system may communicate a read/writerequest to a connected storage system over network channel through anetwork adaptor. If the data is in a cache of the storage system, i.e.,a read hit, then the data may be returned quickly to the host system.This reduces the delay in returning requested data to a host I/Orequest. However, if the requested data is not in the cache of thestorage system, then there may be significant latency realized while thestorage system needs to retrieve the requested data from storage toreturn. Further, the thread or task executing the host read request mayhave to be context switched and deactivated in order to allow the hostsystem to process further I/O requests. When the data is returned to theread request, then the task must be reactivated and data for the taskmust be returned to registers and processor cache to allow processing ofthe returned data for the read request.

In a data mirroring storage environment, a primary storage systemmirrors data to a secondary storage system to provide a backup copy ofthe data in the event of a failure at the primary storage system, sothat access can continue at the secondary storage system.

There is a need in the art for improved techniques for processing hostread/write requests to the cache in a mirrored storage environment inwhich data is being mirrored between a primary and secondary storagesystems.

SUMMARY

Provided are a computer program product, system, and method forproviding track format information when mirroring updated tracks from aprimary storage system to a secondary storage system. The primarystorage system determines a track to mirror to the secondary storagesystem and determines whether there is track format information for thetrack to mirror that the primary storage system maintains for cachingthe track to mirror in the primary cache. The track format informationindicates a format and layout of data in the track, indicated in trackmetadata for the track. The primary storage system sends the trackformat information to the secondary storage system, in response todetermining there is the track format information and mirrors the trackto mirror to the secondary storage system. The secondary storage systemuses the track format information for the track in the secondary cachewhen processing a read or write request to the mirrored track.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a storage environment.

FIG. 2 illustrates an embodiment of a track format table entry.

FIG. 3 illustrates an embodiment of a cache control block.

FIG. 4 illustrates an embodiment of a cache Least Recently Used (LRU)list of tracks in the cache.

FIG. 5 illustrates an embodiment of a demoted cache Least Recently Used(LRU) list of tracks demoted from the cache.

FIG. 6 illustrates an embodiment of a demoted cache control block.

FIG. 7 illustrates an embodiment of a demoted cache control blockdirectory entry.

FIG. 8 illustrates an embodiment of operations to process a read/writerequest received on a first channel, such as a bus interface.

FIG. 9 illustrates receive an embodiment of operations to process aread/write request received on a second channel, such as a network.

FIGS. 10a, 10b, and 10c illustrate an embodiment of operations to stagea track into the cache.

FIG. 11 illustrates an embodiment of operations to close track metadataand determine a track format code for the track in cache of the closedtrack metadata.

FIG. 12 illustrates an embodiment of a storage environment in which aprimary storage system mirrors data to a secondary storage system.

FIG. 13 illustrates an embodiment of an entry in a cache transfer list.

FIG. 14 illustrates an embodiment of a synchronous copy relationship.

FIG. 15 illustrates an embodiment of an asynchronous copy relationship.

FIG. 16 illustrates an embodiment of operations at the primary storagesystem to transfer track format information with updated tracks beingmirrored from the primary storage system to the secondary storagesystem.

FIG. 17 illustrates an embodiment of operations at the primary storagesystem to transfer track format information with updated tracks beingasynchronously mirrored from the primary storage system to the secondarystorage system.

FIG. 18 illustrates an embodiment of operations at the secondary storagesystem to process received track format information and mirrored updatedtracks from the primary storage system.

FIG. 19 illustrates an embodiment of a computer architecture used withdescribed embodiments.

DETAILED DESCRIPTION

In a storage environment, a host system may first communicate aread/write request to a connected storage system over a fast channel,such as a bus interface, such as the Peripheral Component InterconnectExpress (PCIe) interface. For a read/write request over the fast channelwhich is supposed to complete within a threshold time, the host systemholds the application thread for the read/write request in a spin loopwaiting for the request to complete. This saves processor timeassociated with a context swap, which deactivates the thread andreactivates the thread in response to an interrupt when a response tothe read/write request is received. If the data for the read/writerequest sent on the fast channel is not in cache, then the storagesystem may fail the read/write request and the host system maycommunicate the same read/write request over a storage area network viaa host adaptor, which is slower than processing the I/O request over thebus, e.g., PCIe interface. Communicating the read/write request over thesecond channel requires the host system to perform context switch fromthe task handling the read/write request to another task while waitingfor the read/write request to complete. Context switching is costlybecause it requires the processor running the task to clear allregisters and L1 and L2 caches for the new task, and then whencompleting the new task, reactivate the context switched task and returnthe state data to the registers and L1 and L2 caches for the task thatwas context switched while waiting for the read/write request tocomplete.

Certain read/write operations need to be completed within a thresholdtime, else they are failed. The storage system will have to access trackmetadata to process a request to a track. The track metadata providesinformation on the format of data and layout of records in the trackthat are needed in order to perform reads and writes to the track.However, the reading of the track metadata from the storage comprises asubstantial portion of the latency in processing read/write request.Described embodiments provide improvements to cache technology thatreduce cache operation latency by including a track format code in thecache control block for a track in the cache. This track format code maybe used for fast access to the track format from a track format tablewithout having to read the track metadata from storage. By eliminatingthe need to read the track metadata from a metadata track in storage todetermine the track layout, described embodiments increase thelikelihood that read/write requests on the first channel that need to becompleted within a threshold time are completed by accessing the tracklayout information for a track from the track format table, associatingtrack format codes with track format information for common trackformats.

Described embodiments provide further improvements to cache computertechnology by reducing cache latency for a track staged into cache thatwas previously demoted by saving track format metadata, such as thetrack format code, when a track is demoted from the cache. When thedemoted track is later staged into cache, the track format metadata maybe quickly determined by the track format information saved with thedemoted track and included in a cache control block for the staged trackwithout having to read the metadata for the staged track. Avoiding theneed to read the track metadata for a staged track substantially reducesthe latency and delays in staging a track and processing a read/writerequest to the staged track.

With described embodiments, a read/write request to a target track on achannel requiring that the request be completed within a threshold timeis processed if the track format code for the target track is within thecache control block for the target track. Using the track format code toaccess the track format metadata from the track format table reduces thelatency of access to the track format metadata to allow the read/writerequest to complete within the time threshold. This keeps the time thehost thread is spinning on the read/write request task for theread/write request sent on the bus interface within an acceptable timethreshold. However, if the cache control block for the target track doesnot have a valid track format code, then the read/write request on thefirst channel is failed because it is unlikely the read/write requestcan complete within the threshold time given that the track formatmetadata will have to be retrieved from the storage. Failing theread/write request on the first channel, causes the host to redrive theread/write request on the second channel. The processing of theread/write request on the second channel reads in the track metadatafrom the storage to allow for processing the read/write request andadding the track format code to the cache control block for the targettrack.

In a mirror copy storage environment, the primary storage system mirrorsupdated tracks at the primary storage system to the secondary storagesystem. In the prior art when mirroring data, the secondary storagesystem would have to read and access track metadata from a secondarystorage to determine the track format and layout, which introduceslatency when processing I/O requests to updated tracks in a secondarycache at the secondary storage system. Described embodiments reduce thelatency of I/O requests redirected to a secondary storage system afterupdated tracks are mirrored by having the primary storage systemmirroring data transfer track format information of tracks in the cacheat the primary storage system as part of the mirroring. The secondarysystem may maintain this transferred track format information for tracksin its cache, such as track format codes, to use for subsequent I/Orequests to the tracks after the failover. In this way, when subsequentrequests are received after failover to the secondary storage system,the secondary storage system may use the track format informationtransferred over from the primary storage system to determine the trackformat and layout without having to read track metadata from the storageat the secondary storage system. This substantially reduces latency inprocessing I/O requests to mirrored updated tracks at the secondarystorage system because the secondary storage system may avoid accessingand reading the track metadata from the secondary storage. Further,because the track format information is transferred for those tracksthat were updated in the cache of the primary storage, if there is afailover from the primary storage system to the secondary storagesystem, there is a high probability that there will be further requeststo those tracks in the cache at the secondary storage system after thefailover. This means that the latency reduction benefits of transferringthe track format information will likely be realized at the secondarystorage system for multiple I/O requests in the event of a failover.

FIG. 1 illustrates an embodiment of a storage environment in which ahost 100 directs read and write requests to a storage system 102 toaccess tracks in volumes configured in storage devices 104 in a diskenclosure 106. The host 100 includes a processor complex 108 of one ormore processor devices and a memory 110 including an operating system111 executed by the processor complex 108. The host operating system 111generates read and write requests to tracks configured in the storagedevices 104. The host 100 includes hardware to communicate read andwrite requests on two different channels. A first channel is a businterface, such as a Peripheral Component Interconnect Express (PCIe),including a bus 112, a bus switch 114 to connect one or more devices onthe bus 112, including the processor complex 108, a memory system 110,and a bus host adaptor 116 to extend the bus interface over an externalbus interface cable 118 to the storage system 102. Additional businterface technology to extend the bus interface may be utilized,including PCIe extender cables or components, such as a distributed PCIeswitch, to allow PCIe over Ethernet, such as with the ExpEthertechnology. A second channel to connect the host 100 and storage system102 uses a network host adaptor 120, connected to the bus 112, thatconnects to a separate network 122 over which the host 100 and storagesystem 102 additionally communicate. The first channel through the businterface may comprise a faster access channel than the network 122interface through the network host adaptor 120.

The storage system 102 includes a bus interface comprising a bus 124 a,124 b, a bus switch 126 to connect to endpoint devices on the bus 124 a,124 b, and a bus host adaptor 128 to connect to the external businterface cable 118 to allow communication over the bus interface to thehost 100 over the first channel. The storage system 102 includes anInput/Output bay 130 having the bus host adaptor 128, one or more deviceadaptors 132 to connect to the storage devices 104, and one or morenetwork host adaptors 134 to connect to the network 122 and hostsystems.

The storage system 102 includes a processor complex 136 of one or moreprocessor devices and a memory 138 having a cache 140 to cache tracksaccessed by the connected hosts 100. The memory 138 includes a cachemanager 142 and a storage manager 144. The storage manager 144 managesaccess requests from processes in the hosts 100 and storage system 102for tracks in the storage 104. The devices 136, 138, 128, 132, and 134connect over the bus interface implemented in the bus lanes 124 a, 124 band bus switch 126.

The cache manager 142 maintains accessed tracks in the cache 140 forfuture read access to the tracks to allow the accessed tracks to bereturned from the faster access cache 140 instead of having to retrievefrom the storage 104. Further, tracks in the cache 140 may be updated bywrites. A track may comprise any unit of data configured in the storage104, such as a track, Logical Block Address (LBA), etc., which is partof a larger grouping of tracks, such as a volume, logical device, etc.

The cache manager 142 maintains cache management information 146 in thememory 138 to manage read (unmodified) and write (modified) tracks inthe cache 140. The cache management information 146 may include a trackformat table 200 having track format codes for common track formatdescriptors providing details of a layout and format of data in a track;track index 148 providing an index of tracks in the cache 140 to cachecontrol blocks in a control block directory 300; and a Least RecentlyUsed (LRU) list 400 for tracks in the cache 140. The control blockdirectory 300 includes the cache control blocks, where there is onecache control block for each track in the cache 140 providing metadataon the track in the cache 140. The track index 148 associates trackswith the cache control blocks providing information on the tracks in thecache 140. Upon determining that the cache LRU list 400 is full or hasreached a threshold level, tracks are demoted from the LRU list 400 tomake room for more tracks to stage into the cache 140 from the storage104.

In certain embodiments, there may be multiple hosts 100 that connect tothe storage system 102 over the first and second channels to accesstracks in the storage devices 104. In such case, the storage system 102would have at least one bus host adaptor 128 to connect to the businterface 118 of each connected host 100 and one or more network hostadaptors 134 to connect to the network host adaptors 120 on the hosts100.

In one embodiment, the bus interfaces 112, 114, 116, 118, 124 a, 124 b,126, and 128 may comprise a Peripheral Component Interconnect Express(PCIe) bus interface technology. In alternative embodiments, the businterfaces 112, 114, 116, 118, 124 a, 124 b, 126, and 128 may utilizesuitable bus interface technology other than PCIe. The bus host adaptors116 and 128 may comprise PCIe host adaptors that provide the interfaceto connect to the PCIe cable 118. The network 122 may comprise a StorageArea Network (SAN), a Local Area Network (LAN), a Wide Area Network(WAN), the Internet, an Intranet, etc., and the network host adaptors120, 134 provide the network 122 connections between the hosts 100 andstorage system 102.

The storage system 102 may comprise a storage system, such as theInternational Business Machines Corporation (IBM®) DS8000® and DS8880storage systems, or storage controllers and storage systems from othervendors. (IBM and DS8000 are trademarks of International BusinessMachines Corporation throughout the world). The host operating system111 may comprise an operating system such as Z Systems Operating System(Z/OS®) from International Business Machines Corporation (“IBM”) orother operating systems known in the art. (Z/OS is a registeredtrademark of IBM throughout the world).

The storage devices 104 in the disk enclosure 106 may comprise differenttypes or classes of storage devices, such as magnetic hard disk drives,solid state storage device (SSD) comprised of solid state electronics,EEPROM (Electrically Erasable Programmable Read-Only Memory), flashmemory, flash disk, Random Access Memory (RAM) drive, storage-classmemory (SCM), etc., Phase Change Memory (PCM), resistive random accessmemory (RRAM), spin transfer torque memory (STT-RAM), conductivebridging RAM (CBRAM), magnetic hard disk drive, optical disk, tape, etc.Volumes in a storage space may further be configured from an array ofdevices, such as Just a Bunch of Disks (JBOD), Direct Access StorageDevice (DASD), Redundant Array of Independent Disks (RAID) array,virtualization device, etc. Further, the storage devices 104 in the diskenclosure 106 may comprise heterogeneous storage devices from differentvendors and different types of storage devices, such as a first type ofstorage devices, e.g., hard disk drives, that have a slower datatransfer rate than a second type of storage devices, e.g., SSDs.

FIG. 2 illustrates an embodiment of a track format table entry 200, inthe track format table 200, which includes a track format code 202 andthe track format metadata 204. In certain embodiments Count Key Data(CKD) track embodiments, the track format metadata 204 may comprise atrack format descriptor (TFD) indicating a number of records in thetrack, a block size, a number of blocks in the track, a data length ofeach of the records, and a control interval size indicating an amount ofdata that is read or written atomically as a unit, number of blocks in acontrol interval, and whether a control interval spans two tracks, andother information. The track format code 202 may comprise an index valueof the index entry 200 _(i) in the track format table 200. For instance,if there are 32 track format table entries 200 _(i), then the trackformat code 202 may comprise 5 bits to reference the different possiblenumber of 32 entries 200 _(i).

FIG. 3 illustrates an embodiment of a cache control block 300 _(i) forone of the tracks in the cache 140, including, but not limited to, acache control block identifier 302, such as an index value of the cachecontrol block 300 _(i); a track ID 304 of the track in the storage 104;the cache LRU list 306 in which the cache control block 300 _(i) isindicated; an LRU list entry 308 at which the track is indicated; acache timestamp 310 indicating a time the track was added to the cache140 and indicated on the LRU list 306; additional track metadata 312typically maintained for tracks stored in the cache 140, such as a dirtyflag indicating whether the track was modified; a track format code 314comprising one of the track format codes 202 of the track formatmetadata 204 describing the layout of data in the track 304 representedby the cache control block 300 _(i); a track format code valid flag 316indicating whether the track format code 314 is valid or invalid; and aninvalid reason 318 indicating a reason for the track format code validflag 316 code being invalid, as indicated in the track format code validflag 316.

FIG. 4 illustrates an embodiment of an LRU list 400, such as having amost recently used (MRU) end 402 identifying a track most recently addedto the cache 140 or most recently accessed in the cache 140 and a leastrecently used (LRU) end 404 from which the track identified at the LRUend 404 is selected to demote from the cache 140. The MRU end 402 andLRU end 404 point to track identifiers, such as a track identifieraddress or a cache control block index for the track, of the tracks thathave been most recently added and in the cache 140 the longest,respectively, for tracks indicated in that list 400.

FIG. 5 illustrates an embodiment of the demoted cache LRU list 500,having a most recently used (MRU) end 502 identifying a demoted trackmost recently added to the demoted cache LRU list 500 and a leastrecently used (LRU) end 504 from which the demoted track identified atthe LRU end 504 is selected to demote from the demoted cache LRU list500.

FIG. 6 illustrates an embodiment of a demoted cache control block 600,having a track identifier (ID) 602 of a demoted track; a track formatcode 604 if available of the track format metadata 204 in the trackformat table 200 of the demoted track; a pointer to a previous LRU entry606 of a previous demoted track in the demoted cache LRU list 500; apointer to a next LRU entry 608 of a next demoted track in the demotedcache LRU list 500; and a pointer 610 to next demoted cache controlblock in the entry in the demoted cache control block directory 700 forthe track ID 602.

In additional embodiments, the track format code 604 may comprise trackformat information other than a code 604 in a track format table 200,such as other information that may be used to identify or represent thetrack format metadata and layout of data in the tracks and comprisessubstantially less bits of information than the represented track formatmetadata and layout information.

FIG. 7 illustrates an embodiment of an entry 700 _(i) in the demotedcache control block directory 700 that includes pointers 702 ₁, 702 ₂ .. . 702 _(n) to demoted cache control blocks 600 _(i). Each demotedcache control block 600 _(i) maps to one entry in the demoted cachecontrol block directory 700 based on the track ID 602. In oneembodiment, a hash function of the track identifier would produce theentry in the demoted cache control block directory 700 in which thedemoted cache control block 600 _(i) is indicated.

FIG. 8 illustrates an embodiment of operations performed by the cachemanager 142 and storage manager 144 to process a read/write request to atarget track received on a first fast channel, such as the PCIe businterface via bus host adaptor 128. Upon receiving (at block 800) theread/write request at the bus host adaptor 128, if (at block 802) thetarget track is not in the cache 140, then the storage manager 144returns (at block 804) fail to the read/write request on the firstchannel or bus host adaptor 128 to the host 100, which causes the host100 to retry the read/write request on the second channel or networkhost adaptor 120, 134. Failure is returned because if the target trackis not in the cache 140, then the target track and track metadata needsto be staged into cache 140, which would likely exceed the timethreshold for read/writes on the first channel, where the host processoris spinning on the thread of the read/write request. If (at block 802)the target track is in the cache 140 is a write and if (at block 808)the write modifies the track format, then the cache manager 142 sets (atblock 810) the track format code valid flag 316 to invalid and indicates(at block 812) the invalid reason 318 that the track in the cache 140was invalidated as track format change. The storage manager 144 thenreturns (at block 804) fail to the host 100 because the track metadataneeds to be read from the storage 104 to update with the modified trackformat.

If (at block 806) the read/write request is a read or if (at block 808)the request is a write that does not modify the track format, then thecache manager 142 determines (at block 814) if the track format codevalid flag 316 is set to valid. If so, then the cache manager 142determines (at block 816) the track format metadata 204 in the trackformat table 200 corresponding to the track format code 314 in the cachecontrol block 300 _(i). The cache manager 142 uses (at block 818) thetrack format layout indicated in the determined track format metadata204 to process the read or write request to the target track in thecache 140. If the request is a write, a dirty flag 312 in the cachecontrol block 300 _(i) may be set to indicate the track is modified. If(at block 814) the track format code valid flag 316 is invalid, meaningthere is no fast access to track format information available throughthe track format code 314, then the storage manager 144 returns (atblock 804) fail on the bus interface to the bus host adaptor 128 becausethe track format table 200 cannot be used, and the track metadata needsto be read from the storage 104, which would introduce too much latencyfor the fast read/write on the first channel.

With the embodiment of operations of FIG. 8, during a fast write overthe bus interface or first channel, if the track format metadata may beaccessed without latency through the track format table 200, then theread/write request is allowed to proceed when the transaction can beprocessed very quickly because the track metadata can be obtaineddirectly from the track format table 200 through the track format code314, without having to read the track metadata from storage 104.However, if the cache control block 300 _(i) does not have a valid trackformat code 314 to allow low latency access of track format metadata,then the read/write request is failed because the transaction will notlikely complete within a fast time threshold. This determination isimportant to avoid host delays in processing other tasks while the hostprocessor is spinning on the thread handling the read/write requestwhile waiting for the read/write request to complete. If the trackmetadata can be accessed from the track format table 200 than there is ahigh likelihood the read/write can complete on the bus interface channelwithin the time required to avoid the host processor holding the threadfor too long, which causes other I/O requests to be queued and delayed.If the track metadata cannot be accessed from the track format table 200and needs to be read from the storage 104, then it is unlikely theread/write request will complete within the time threshold for the hostprocessor to spin on the thread for the read/write request, and failureis returned. Returning failure when the track metadata cannot beobtained from the track format table 200 causes the host thread waitingon the read/write request task to be deactivated and the host processormay context switch to processing other tasks, and then the read/writerequest is retried on the second network channel during the contextswitch.

FIG. 9 illustrates an embodiment of operations performed by the cachemanager 142 and storage manager 144 to process a read/write request to atarget track received on a second channel, such as the network 122 onnetwork host adaptor 134. Upon receiving (at block 900) the read/writerequest, if (at block 902) the target track is not in the cache 140,then the cache manager 142 proceeds (at block 904) to block 1000 in FIG.10a to stage the track into the cache 140. If (at block 908) theread/write request is a write and if (at block 910) the write modifiesthe track format, then the cache manager 142 updates (at block 912) thetrack metadata to indicate the modified track format and sets (at block914) the track format code valid flag 316 to invalid. The track metadata312 is further updated (at block 916) to indicate the track is modifiedor dirty. If (at block 908) the request is a read or from block 916, thecache manager 142 uses (at block 918) the track format layout indicatedin the track format metadata to process the read or write request to thetarget track in the cache 140.

If (at block 902) the target track is in the cache 140 and if (at block930) the track format code valid flag 316 is set to valid, then thecache manager 142 determines (at block 932) the track format metadata204 in the track format table 200 corresponding to the track format code314 in the cache control block 300 _(i) for the target track. From block932, control proceeds to block 908 to process the read/write request. If(at block 930) the track format code valid flag 316 is set to invalid,then the cache manager 142 reads (at block 934) the track metadata forthe target track from the storage 104 to determine the track format,e.g., size of blocks, control interval, layout of records on the track,etc. From block 934, control proceeds to block 908 to process theread/write request.

With the embodiment of FIG. 9, when the read/write request is receivedon the second slower channel, such as over the network 122, where thehost operating system 111 would have performed a context switch for thethread handling the read/write request, the cache manager 142 may readthe track metadata from the storage 104 to determine the track layout toprocess the request. During this time, the host processing of furtherhost requests is not delayed because the host thread handling theread/write request is context switched and not active, until theread/write request returns complete.

FIGS. 10a, 10b, and 10c illustrate an embodiment of operations performedby the cache manager 142 to stage a track into the cache 140, which maybe invoked at block 904 in FIG. 9 when the target track of a read/writerequest is not in the cache 140. Upon initiating (at block 1000) theoperation to stage a track into the cache 140, if (at block 1002) thecache LRU list 400 is full, then the track at the LRU end 404 of thecache LRU list 400 is selected (at block 1004) to demote. If (at block1006) the demoted cache LRU list 500 is full, then the cache manager 142selects (at block 1008) a demoted track indicated at the LRU end 504 ofthe demoted cache LRU list 500 to demote. The selected demoted track isremoved (at block 1010) from the LRU end 504. The cache manager 142adjusts (at block 1012) a demoted cache control block 600 _(j) whosepointer to next LRU entry 608 points to the selected demoted track inthe demoted cache LRU list 500 to indicate that the pointer 608 is null,because now that entry is at the LRU end 504 when the selected demotedtrack is removed from the demoted cache LRU list 500.

The cache manager 142 determines (at block 1014) an entry 700 _(i) inthe demoted cache control block directory 700 having the demoted cachecontrol block 600 _(S) of the selected demoted track. In one embodiment,the entry 700 _(i) may be determined by applying a hash function to theselected demoted track identifier. The hash function may map any of thetrack identifiers in the storage 104 to one of the entries 700 _(i) inthe demoted cache control block directory 700. The cache manager 142then needs to adjust the pointer 610 that points to the demoted cachecontrol block 600 _(S) of the selected demoted track. For this, thecache manager 142 adjusts (at block 1016) a demoted cache control block600 _(j) in the determined entry 700 _(i) that points to the demotedcache control block 600 _(S) of the selected demoted track to point tothe demoted cache control block 600 _(k) pointed to by the pointer 610of the demoted cache control block 600 _(S) of the selected demotedtrack. The demoted cache control block 600 _(S) for the selected demotedtrack is deleted (at block 1018) and indication of the deleted demotedcache control block 600 _(S) is removed (at block 1020) from the entry700 _(i) in the demoted cache control block directory 700.

From block 1020 or if (at block 1006) the demoted cache LRU list 500 isnot full, control proceeds (at block 1022) to block 1030 in FIG. 10b toadd the demoted track from the cache 140 to the demoted cache LRU list500. Upon initiating (at block 1030) the operation to add the demotedtrack to the demoted cache LRU list 500, the cache manager 142 indicates(at block 1032) the demoted track at the MRU end 502 of the demotedcache LRU list 500. The cache manager 142 determines (at block 1034) thecache control block 300 _(DT) for the demoted track from the cache 140and the track format code 314 for the demoted track if one is included.The cache manager 142 generates (at block 1036) a demoted track cachecontrol block 600 _(DT) for the track being demoted indicating a trackidentifier 602 of the demoted track and the determined track format code314 for the demoted track to save the track format code in field 604 ofthe demoted track cache control block 600 _(DT). If there was no validtrack format code 314 in the cache control block 300 _(DT), then a nullvalue may be indicated in the field 604 indicating there is no validtrack format code for the demoted track.

The cache manager 142 determines (at block 1038) the entry 700 _(i) inthe demoted cache control block directory 700 that will be used toindicate the demoted cache control block 600 _(DT) of the demoted track.The next pointer 610 of the last demoted cache control block 600 _(i)indicated in the determined entry 700 _(i) is adjusted (at block 1040)to point to the demoted cache control block 600 _(DT) for the trackbeing demoted. The demoted track control block 600 _(DT) for the demotedtrack is indicated (at block 1042) in the determined entry 700 _(i) inthe demoted cache control block directory 700, e.g., at the end of theentry 700 _(i). After adding the demoted track to the demoted LRU list500 and the demoted cache control block 600 _(DT) to the demoted cachecontrol block directory 700, control proceeds (at block 1044) to block1060 in FIG. 10c to stage the target track to the cache 140. Controlalso proceeds to block 1060 in FIG. 10c to stage the track if (at block1002 in FIG. 10a ) the cache LRU list 400 is not full, so that a trackdoes not need to be demoted from the cache 140 and added to the demotedcache LRU list 500.

Upon initiating (at block 1060) the operation to stage the track to thecache 140, the cache manager 142 stages (at block 1062) the target trackinto the cache 140 from the storage 104. The staging of the actual trackdata from the storage 104 may have been initiated earlier before orduring the operations of FIGS. 10a, 10b to manage the demoted cache LRUlist 500 and demoted cache control block directory 700. The target trackstaged into the cache 140 is indicated (at block 1064) at the MRU end402 of the cache LRU list 400. A cache control block 300 _(ST) isgenerated (at block 1066) for the staged track. Control then proceeds toblock 1068 to determine if there is a demoted cache control block 600_(ST) for the staged track that has a track format code 604 (or othertrack format information) that can be included in the cache controlblock 300 _(ST) created for the staged track. A determination is made(at block 1068) of the entry 700 _(i) in the demoted cache control blockdirectory 700 that could have a demoted cache control block 600 _(ST)for the staged track, which entry 700 _(i) may be determined by applyinga hash function to the track identifier of the staged track. The firstdemoted cache control block 600 _(SEL) in the determined entry 700 _(i)is selected (at block 1070). If (at block 1072) the track identifier 602of the selected demoted cache control block 600 _(SEL) matches the trackidentifier of the staged track, then the track format code 604 in theselected demoted cache control block 600 _(SEL) is included (at block1074) in the cache control block 300 _(ST) for the staged track. Thecache manager 142 may then perform (at block 1076) the operations atblocks 1010, 1016, 1018, and 1020 in FIG. 10a to remove demoted trackinformation for the staged track, including removing the demoted cachecontrol block 600 _(ST) for the staged track, removing the staged trackfrom the demoted cache LRU list 500, removing the indication of thedemoted cache control block 600 _(ST) from the demoted cache controlblock directory 700, and adjusting pointers 606, 608, 610 in otherdemoted cache control blocks 600 _(i) that pointed to the demoted trackor demoted cache control block 600 _(ST) for the staged track becausethe staged track is no longer demoted but active in cache 140.

If (at block 1072) the selected demoted cache control block 600 _(SEL)is not for the staged track and if (at block 1078) there is a nextdemoted cache control block 600 _(i) in the entry 700 _(i), which may beindicated in the pointer 610, then that next demoted cache control blockis selected (at block 1080) and control returns to block 1070 todetermine whether this next demoted cache control block 600 _(i) is forthe staged track. If (at block 1078) there are no further next demotedcache control blocks in the determined entry 700 _(i) to consider, thenthe track format code 202 from a demoted track information cannot beused and the cache manager 142 reads (at block 1082) the metadata forthe track from the storage 104 to determine the track format. From block1076 after using the track format code 604 from the demoted cachecontrol block for the staged track or after reading (at block 1082) themetadata for the staged track, control returns (at block 1084) to block904 in FIG. 9 with staging complete to perform the read/write operationwith respect to the staged track.

With the embodiments of FIGS. 10a, 10b, and 10c , the track format codefor a track demoted from cache can be saved and later used when thedemoted track is staged back into cache. This allows the track metadataformat to be quickly determined for the demoted track staged back intocache without having to read the metadata for the track from storage.The computer performance for cache operations, particularly staging, aresubstantially improved and latency reduced by determining the trackmetadata format and layout of a track being staged into cache withouthaving to read the metadata for the track.

FIG. 11 illustrates an embodiment of operations performed by the cachemanager 142 when closing the track metadata for a track in the cache140, which involves destaging the track metadata to the storage 104 ifchanged. Upon closing (at block 1100) the track metadata for a track inthe cache 140, the cache manager 140 processes (at block 1102) the trackmetadata to determine a track format or a layout of data in the track.If (at block 1104) the track format table 200 does not have a trackformat 204 matching the determined track format from the track metadata,which may happen if the determined track format is irregular, then thetrack format code valid flag 316 is set (at block 1106) to invalid andthe invalid reason 318 is set to indicate that the track format is notsupported. In such situation, read/write requests to the track having anirregular format are only processed when received through the secondchannel via network host adaptor 134.

If (at block 1104) the track format table has a track format 204matching the determined track format from the track metadata, then thecache manager 142 determines the track format code 202 for thedetermined track format 204 in the track format table 200 and includesthe track format code 202 in the field 314 in the cache control block300 _(i). The track format code valid flag 316 is set (at block 1116) tovalid. From block 1108 or 1116, control proceeds to block 1118 todestage the track metadata from the memory 138 if modified or discard ifnot modified.

With the operations of FIG. 11, the track format information may beindicated in the cache control block 300 _(i) with a track format code202 having a limited number of bits to index track format metadata 204describing track layout in a track format table 200, where the trackmetadata itself would not fit into the cache control block 300 _(i). Forfuture read/write accesses, if a valid track format code 314 isprovided, then the cache manager 142 may use that code 314 to obtainwith low latency the track format metadata 204 from the track formattable 200 without having to read the track metadata from the storage 104and process to determine the track format.

Storage Mirroring Environment

FIG. 12 illustrates an embodiment of the storage environment of FIG. 1of a host 1200 having components 1208, 1210, 1211, 1212, 1214, 1216,1220 that comprise the components 108, 110, 111, 112, 114, 116, 120 ofthe host 100 described with respect to FIG. 1. The embodiment of FIG. 12additionally includes a primary storage system 1202 ₁ and a secondarystorage system 1202 ₂, where each of the primary 1202 ₁ and secondary1202 ₂ storage systems include the components of the storage system 102as described with respect to FIG. 1. Further, the host 1200 may includeone or more bus host adaptors 1216 having links 1218 ₁ and 1218 ₂ to theprimary 1202 ₁ and secondary 1202 ₂ storage systems, where the links1218 ₁ and 1218 ₂ may comprise a PCIe over Ethernet or network type linkto extend the PCIe link over a network, such as an Ethernet network,such as the ExpEther technology. The hosts 1200 may also communicatewith the primary 1202 ₁ and secondary 1202 ₂ storage systems over thenetwork 1222 via network host adaptors 1220 and 134.

Each of the primary 1202 ₁ and secondary 1202 ₂ storage systems wouldinclude, as shown and described with respect to FIGS. 1-7, a processorcomplex 136 and the components in the memory 138, including components140, 142, 144, 146, 148, 200, 300, 400, 500, 600, and 700. Further, eachof the primary 1202 ₁ and secondary 1202 ₂ storage systems would becapable of performing the operations of FIGS. 8-9, 10 a, 10 b, 10 c, and11 to manage their respective cache 140 and I/O requests directedthereto.

Each of the primary 1202 ₁ and secondary 1202 ₂ storage systemsadditionally include a cache transfer list 1300 ₁, 1300 ₂ that includestracks and track format codes to transfer to the other system 1202 ₂,1202 ₁ to provide the secondary storage system track format formation; areplication manager 1252 ₁, 1252 ₂ to replicate data to the other system1202 ₂, 1202 ₁, respectively; synchronous copy relationships 1400 ₁,1400 ₂ indicating source storage and a target storages in relationshipssuch that writes to a source storage in a synchronous copy relationshipneed to be synchronously copied to the target storage before returningcomplete; and asynchronous copy relationships 1500 ₁, 1500 ₂ indicatingsource storages and target storages in asynchronous copy relationships,such that writes to the source storage are asynchronously copied to thetarget storage, and may be copied in consistency group such that themirrored data is consistent as of a point-in-time.

The primary and secondary replication managers 1252 ₁, 1252 ₂ may createactive copy relationships 1400 ₁, 1400 ₂, 1500 ₁, 1500 ₂ to manage themirroring of data from the primary storage 104 ₁ to the secondarystorage system storage 104 ₂ and vice versa.

The primary 1202 ₁ and secondary 1202 ₂ storage systems may eachcomprise an enterprise storage controller/server suitable for managingaccess to attached storage devices, such as, but not limited to, theInternational Business Machines Corporation's (“IBM”) DS8000® storagesystem or other vendor storage servers known in the art. (DS8000 is aregistered trademark of IBM in countries throughout the world).

In one embodiment, the replication managers 1252 ₁, 1252 ₂ may comprisesa program for managing the mirroring of volumes across systems, such as,but not limited to, the IBM mirroring programs Geographically DispersedParallel Sysplex® (GDPS)®, and Tivoli® Storage Productivity Center forReplication (TPC-R) that define a replication session and copy pairs.Different types of techniques may be selected to copy the data, such assynchronous mirroring, asynchronous mirroring or point-in-time copying,or combinations of multiple of these different mirroring types.

During operations while both primary 1202 ₁ and secondary 1202 ₂ storagesystems are operating, the bus host adaptor 1228 and network hostadaptor 1234 would direct a received I/O request to the primary storagesystem 1202 ₁, and the replication manager 1252 ₁ of the primary storagesystem 1202 ₁ would mirror data at the primary storage 104 ₁ to thesecondary storage system 1202 ₂.

FIG. 13 illustrates an embodiment of an entry 1300 _(i,j) in the cachetransfer lists 1300 ₁, 1300 ₂, where entry j for storage system i, oneof storage systems 1202 ₁, 1202 ₂, includes a track identifier (ID) 1302and a track format code 1304 comprising one of the track format codes202 in the track format table 202 indicating track format metadata 204for the track 1302.

FIG. 14 illustrates an instance of a synchronous copy relationship 1400_(i,j), where storage system i has an instance j of a synchronous copyrelationship including a copy identifier 1402; a source storage 1404,e.g., a primary volume, in a primary storage at the primary storagesystem 1202 _(i) from which data is synchronously copied; and a targetstorage 1406, e.g., secondary volume, in a secondary storage of thesecondary storage system 1202 ₂ to which tracks in the source storage1404 are copied. For a synchronous copy relationship, the primarystorage system 1202 ₁ receiving the write request does not returncomplete until acknowledgment is received from the secondary storagesystem 1202 ₂ that the write data is copied to the secondary storage 140₂, which may be returned when the write data is written to a secondarycache 140 ₂ at the secondary storage system 1202 ₂ or when the writedata is written to the secondary storage 104 ₂.

FIG. 15 illustrates an embodiment of an instance of an asynchronous copyrelationship 1500 _(i,j), where storage i has an instance j of a copyrelationship including a copy identifier 1502 identifying anasynchronous copy created by the replication manager 1252 ₁, 1252 ₂;source storage 1506, e.g., primary volume, at the primary storage system1202 ₁, from which data is copied; target storage 1508, e.g., asecondary volume, at the second storage system 1202 ₂ to which tracks inthe source storage 1506 are copied; an out-of-synch data structure 1510,such as a bitmap, indicating tracks or other data units in the sourcestorage 1506 that have been changed and need to be copied to the targetstorage 1508; and a change recording data structure 1512, such as achange recording bitmap, indicating source storage 1506 tracks that havebeen updated or changed to be included in the next consistency group,i.e., out-of-synch data structure 1510, to form.

To create a new consistency group, the change recording data structure1512, indicating data updated while copying changed data for the currentconsistency group being formed, is merged with the out-of-synch datastructure 1510, including changed data being copied for the currentconsistency group. After the merging, the change recording datastructure 1512 is cleared to record new updates for a next consistencygroup while the data indicated as changed in the out-of-synch datastructure 1510 is being copied to the target storage 1508 to form a newcurrent consistency group at the target storage 1508.

FIG. 16 illustrates an embodiment of operations performed by thereplication manager 1252 ₁, 1252 ₂ to mirror updated tracks at theprimary storage system 1202 ₁ to the secondary storage system 1202 ₂,for updated tracks in a synchronous 1400 _(1,j) or asynchronous 1500_(1,j) relationship. FIGS. 16, 17, and 18 are described with respect tomirroring from the primary storage system 1202 ₁ to the secondarystorage system 1202 ₂. However, the operations may also apply withrespect to mirroring data from the secondary storage system 1202 ₂ tothe primary storage system 1202 ₁ as part of copy relationships 1400_(2,j) and 1500 _(2,j) maintained at the secondary storage system 1202₂. The mirroring operations performed at blocks FIGS. 16, 17, and 18 maybe performed with respect to updated tracks resulting from writeoperations to the primary storage system 1202 ₁, which may occur duringor after an initial synchronization to copy all of the tracks from thesource storage to the target storage. Alternatively, the mirroring ofthe write operation may be performed for a point-in-time copy orsnapshot that copies over changed data to be consistent as of apoint-in-time. In FIGS. 16, 17, and 18, the replication managers 1252 ₁,1252 ₂ may interact with the cache manager 142 ₁, 142 ₂ to performoperations with respect to cache control blocks 300 _(i) for tracks inthe primary cache 140 ₁ and secondary cache 140 ₂, respectively.

Upon (at block 1600) initiating at the primary storage system 1202 ₁ anoperation to mirror updated track or tracks, the replication manager1252 ₁ determines (at block 1602) an updated track or tracks in theprimary cache to mirror to the secondary storage system 1202 ₂, whichmay be mirrored as part of a synchronous 1400 _(1,i) or asynchronous1500 _(1,i) copy relationship. If (at block 1604) the cache controlblock 300 _(i) for the updated track has valid track format information,such as a track format code 314 indicated as valid in the track formatcode valid flag 316, and if (at block 1606) the updated track does notchange the format of the track, then the replication manager 1252 ₁transfers (at block 1608) the track format information, i.e., trackformat code 314, to the secondary storage system 1202 ₂. The updatedtrack is also transferred (at block 1610) to the secondary storagesystem 1202 ₂.

In a synchronous copy operation, the track format code 314 may betransferred in a message to the secondary storage system 1202 ₂ beforesending the updated track. Alternatively, the track format code 314 maybe sent with the updated track. With a synchronous copy operation, thewrite operation is not completed until the secondary storage system 1202₂ acknowledges the updated track was received.

If (at block 1604) the cache control block 300 _(i) for the updatedtrack does not have a valid track format code, i.e., the track formatcode valid flag 316 indicates there is no valid track format code 314,then the replication manager 1252 ₁ transfers (at block 1612) theupdated track to the secondary storage system 1202 ₂ without trackformat information. If no track format information is provided, then thesecondary storage system 1202 ₂ needs to determine the track formatinformation from track metadata for the track or rebuild the trackformat information. If (at block 1606) the update to the track modifiesthe format or layout of the track, which would render outdated the trackformat code 314 maintained for the track, then the replication manager1252 ₁ indicates (at block 1614) the track format code 314 as invalid,such as by setting the track format code valid flag 316 to invalid andproceeds to block 1612 to transfer the updated track without trackformat information.

With the embodiment FIG. 16, when mirroring data to a secondary storagesystem 1202 ₂, the primary storage system 1202 ₁ sends to the secondarystorage system 1202 ₂ the abbreviated track format information to beable to use for the mirrored updated tracks in the secondary cache 140₂, and thus avoid having to read in the track metadata to determine thetrack format or layout.

FIG. 17 illustrates an embodiment of operations performed by thereplication manager 1252 ₁ to transfer track format information whenperforming an asynchronous mirror operation for an asynchronous copyrelationship 1500 _(1,i) to mirror updated tracks in the primary cache140 ₁ indicated in an out-of-synch data structure 1510. Upon initiating(at block 1700) an operation to mirror tracks in a consistency groupindicated in an out-of-synch data structure 1510 of an asynchronous copyrelationship 1500 _(1,i), the replication manager 1252 ₁ performs a loopof operations from blocks 1702 through 1708 for each of the updatedtracks indicated in the out-of-synch data structure 1510. If (at block1704) there is a valid track format code 316 in the cache control block300 _(i) for the updated track, i.e., having a valid track format codeflag 314 indicating the code 316 is valid, then an entry 1300 _(1,i) isadded (at block 1706) to a first cache transfer list 1300 ₁ having thetrack identifier 1302 of the updated track and the track format code1304 from field 314 in the cache control block 300 _(i) for the updatedtrack being considered. After adding the entry 1300 _(1,i) to theprimary cache transfer list 1300 ₁ (at block 1704) or if (at block 1704)there is no valid track format code 314 in the cache control block 300_(i), then control returns (at block 1708) to block 1702 consider a nextupdated track indicated in the out-of-synch data structure 1510 orproceed to block 1710 after all indicated updated tracks are considered.At block 1710, the replication manager 1252 ₁ transmits the primarycache transfer list 1300 ₁ to the secondary storage system 1202 ₂ andcopies (at block 1712) the modified tracks indicated in the out-of-synchdata structure 1510 to the secondary storage system 1202 ₂.

In one embodiment, the cache transfer list 1300 ₁ may be sent to thesecondary storage system 1202 ₂ in a message before or while copying theupdated tracks to the secondary storage system 1202 ₂. In an alternativeembodiment, the track format information, e.g., track format codes 314,may not be transferred in a cache transfer list 1300 ₁ but instead eachtrack format code 314 for an updated track may be sent with the updatedtrack when the updated track is sent.

FIG. 18 illustrates an embodiment of operations performed by thereplication manager 1252 ₂ at the secondary storage system 1202 ₂ toprocess a received mirrored track along with track format information ifprovided. Upon the secondary storage system 1202 ₂ receiving (at block1800) track information, e.g., track format code 314, for a track beingmirrored, the received track format information is saved (at block1802). For synchronous mirroring the track format information maycomprise the track format code 314 for one track being mirrored and forasynchronous mirroring, the received track format information maycomprise a primary cache transfer list 1300 ₁ having track format codes314 for multiple updated tracks that will be mirrored. Upon receiving(at block 1804) a mirrored updated track, which may be received with orafter the track format information, e.g., track format code 202, thereceived updated track is stored (at block 1806) in the secondary cache140 ₂. Further, updated tracks may be received without track formatinformation if not sent from the primary storage system 1202 ₁. If (atblock 1808) there is no cache control block 300 ₁ for the updated trackadded to the secondary cache 140 ₂, then a cache control block 300 _(i)is created (at block 1810) for the updated track. If (at block 1808)there is already a cache control block 300 _(i) for the updated track orafter creating (at block 1810) a cache control block 300 _(i), then if(at block 1814) the updated track does not change the format of thetrack, the replication manager 1252 ₂ determines (at block 1816) whethera track format code 202 was provided for the updated track, such as sentindividually before or with the updated track or in field 1302 in acache transfer list 1300 ₁. If (at block 1816) a track format code 202was provided for the updated track, then the track format code 202 isindicated (at block 1818) in the cache control block 300 _(i) in field314.

If (at block 1816) a track format code was not provided for the updatedtrack, then the replication manager 1252 ₂ reads (at block 1820) thetrack metadata for the track from the secondary storage 104 ₂ anddetermines (at block 1822) the track format code 202 associated withtrack format metadata 204 in the track format table 200 matching that ofthe track format indicated in the accessed track metadata. Thedetermined track format code 202 is indicated (at block 1824) in thecache control block 300 _(i) for the received updated track, and thetrack format valid flag 316 is set to valid.

If (at block 1814) the updated track changes the format of the track,then the secondary cache manager 142 ₂ rebuilds (at block 1826) thetrack metadata from the updated track and determines (at block 1828) atrack format code 202 associated with track format metadata 204 in thetrack format table 200 matching the track format indicated in therebuilt track metadata. The determined track format code 202 isindicated (at block 1824) in field 314 in the cache control block 300_(i).

With the operations of FIG. 18, if the primary storage system 1202 ₁provides the track format information with a mirrored updated track,then the secondary storage system 1202 ₂ includes that provided trackformat information, e.g., track format code 202, in the cache controlblock 300 _(i) for the mirrored track added to the secondary cache 140 ₂to be available to use when the secondary storage system 1202 ₂processes a read or write request to the updated track in the secondarycache 140 ₂. In this way, the secondary storage system 1202 ₂ avoidslatency to access and read track metadata for the track in the secondarycache 140 ₂ from the secondary storage 104 ₂ because the track formatand layout may be determined from the track format information, such astrack format code 316, included in the cache control block 300 _(i) ofthe updated racks.

The present invention may be implemented as a system, a method, and/or acomputer program product. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The computational components of FIGS. 1 and 12, including the host 100,1200 and storage systems 102, 102 ₂, 1202 ₁ and 1202 ₂ may beimplemented in one or more computer systems, such as the computer system1902 shown in FIG. 19. Computer system/server 1902 may be described inthe general context of computer system executable instructions, such asprogram modules, being executed by a computer system. Generally, programmodules may include routines, programs, objects, components, logic, datastructures, and so on that perform particular tasks or implementparticular abstract data types. Computer system/server 1902 may bepracticed in distributed cloud computing environments where tasks areperformed by remote processing devices that are linked through acommunications network. In a distributed cloud computing environment,program modules may be located in both local and remote computer systemstorage media including memory storage devices.

As shown in FIG. 19, the computer system/server 1902 is shown in theform of a general-purpose computing device. The components of computersystem/server 1902 may include, but are not limited to, one or moreprocessors or processing units 1904, a system memory 1906, and a bus1908 that couples various system components including system memory 1906to processor 1904. Bus 1908 represents one or more of any of severaltypes of bus structures, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. By way of example, andnot limitation, such architectures include Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnects (PCI) bus.

Computer system/server 1902 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 1902, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 1906 can include computer system readable media in theform of volatile memory, such as random access memory (RAM) 1910 and/orcache memory 1912. Computer system/server 1902 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 1913 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 1908 by one or more datamedia interfaces. As will be further depicted and described below,memory 1906 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 1914, having a set (at least one) of program modules1916, may be stored in memory 1906 by way of example, and notlimitation, as well as an operating system, one or more applicationprograms, other program modules, and program data. Each of the operatingsystem, one or more application programs, other program modules, andprogram data or some combination thereof, may include an implementationof a networking environment. The components of the computer 1902 may beimplemented as program modules 1916 which generally carry out thefunctions and/or methodologies of embodiments of the invention asdescribed herein. The systems of FIGS. 1 and 12 may be implemented inone or more computer systems 1902, where if they are implemented inmultiple computer systems 1902, then the computer systems maycommunicate over a network.

Computer system/server 1902 may also communicate with one or moreexternal devices 1918 such as a keyboard, a pointing device, a display1920, etc.; one or more devices that enable a user to interact withcomputer system/server 1902; and/or any devices (e.g., network card,modem, etc.) that enable computer system/server 1902 to communicate withone or more other computing devices. Such communication can occur viaInput/Output (I/O) interfaces 1922. Still yet, computer system/server1902 can communicate with one or more networks such as a local areanetwork (LAN), a general wide area network (WAN), and/or a publicnetwork (e.g., the Internet) via network adapter 1924. As depicted,network adapter 1924 communicates with the other components of computersystem/server 1902 via bus 1908. It should be understood that althoughnot shown, other hardware and/or software components could be used inconjunction with computer system/server 1902. Examples, include, but arenot limited to: microcode, device drivers, redundant processing units,external disk drive arrays, RAID systems, tape drives, and data archivalstorage systems, etc.

The terms “an embodiment”, “embodiment”, “embodiments”, “theembodiment”, “the embodiments”, “one or more embodiments”, “someembodiments”, and “one embodiment” mean “one or more (but not all)embodiments of the present invention(s)” unless expressly specifiedotherwise.

The terms “including”, “comprising”, “having” and variations thereofmean “including but not limited to”, unless expressly specifiedotherwise.

The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise.

The terms “a”, “an” and “the” mean “one or more”, unless expresslyspecified otherwise.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Onthe contrary a variety of optional components are described toillustrate the wide variety of possible embodiments of the presentinvention.

When a single device or article is described herein, it will be readilyapparent that more than one device/article (whether or not theycooperate) may be used in place of a single device/article. Similarly,where more than one device or article is described herein (whether ornot they cooperate), it will be readily apparent that a singledevice/article may be used in place of the more than one device orarticle or a different number of devices/articles may be used instead ofthe shown number of devices or programs. The functionality and/or thefeatures of a device may be alternatively embodied by one or more otherdevices which are not explicitly described as having suchfunctionality/features. Thus, other embodiments of the present inventionneed not include the device itself.

The foregoing description of various embodiments of the invention hasbeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseform disclosed. Many modifications and variations are possible in lightof the above teaching. It is intended that the scope of the invention belimited not by this detailed description, but rather by the claimsappended hereto. The above specification, examples and data provide acomplete description of the manufacture and use of the composition ofthe invention. Since many embodiments of the invention can be madewithout departing from the spirit and scope of the invention, theinvention resides in the claims herein after appended.

1. A computer program product for mirroring data from a primary storagesystem having a primary cache and a primary storage to a secondarystorage system having a secondary cache and a secondary storage, thecomputer program product comprising a computer readable storage mediumhaving computer readable program code executed in the primary and thesecondary storage systems to perform operations, the operationscomprising: determining, by the primary storage system, a track tomirror from the primary storage system to the secondary storage system;determining, by the primary storage system, whether there is trackformat information for the track to mirror that the primary storagesystem maintains for caching the track to mirror in the primary cache,wherein the track format information indicates a format and layout ofdata in the track, indicated in track metadata for the track; sending,by the primary storage system, the track format information to thesecondary storage system, in response to determining there is the trackformat information; mirroring, by the primary storage system, the trackto mirror to the secondary storage system; storing, by the secondarystorage system, a mirrored track mirrored from the primary storagesystem in the secondary cache; creating, by the secondary storagesystem, a cache control block for the mirrored track; determining, bythe secondary storage system, whether the primary storage systemprovided track format information for the mirrored track; including, bythe secondary storage system, the track format information in the cachecontrol block for the mirrored track in response to determining that theprimary storage system provided track format information for themirrored track; and using, by the secondary storage system, the trackformat information for the track in the secondary cache when processinga read or write request to the mirrored track.
 2. The computer programproduct of claim 1, wherein the track format information comprises atrack format code defined in a primary track format table at the primarystorage system and a secondary track format table at the secondarystorage system associating track format codes with track formatmetadata.
 3. The computer program product of claim 1, wherein the trackto mirror comprises an update track in the primary cache, wherein theoperations further comprise: determining, by the primary storage system,whether the updated track modifies the format of the track;invalidating, by the primary storage system, the track formatinformation maintained for the track in the primary cache in response todetermining that the updated track modifies the format of the track,wherein the invalidated track format information for the track to mirroris not sent to the secondary storage system when mirroring the updatetrack to the secondary storage system; rebuilding, by the secondarystorage system, track metadata for the updated track modifying theformat of the track; determining, by the secondary storage system, trackformat information from the track metadata rebuilt from the updatedtrack; and including, by the secondary storage system, the determinedtrack format information in a cache control block for the updated trackin the secondary cache. 4-5. (canceled)
 6. The computer program productof claim 1, wherein the track to mirror comprises an updated trackupdated by a write operation, wherein the track to mirror is mirrored tothe secondary storage as part of a synchronous mirror operation wherethe write operation does not complete until the updated track ismirrored to the secondary storage system, wherein the sending the trackformat information comprises: sending, by the primary storage system, amessage to the secondary storage system including the track formatinformation for the updated track before mirroring the updated track tothe secondary storage system.
 7. The computer program product of claim1, wherein the track to mirror comprises an updated track updated by awrite operation, wherein the track to mirror is mirrored to thesecondary storage as part of a synchronous mirror operation where thewrite operation does not complete until the updated track is mirrored tothe secondary storage system, wherein the track format information issent to the secondary storage system with the mirroring of the updatedtrack.
 8. The computer program product of claim 1, wherein theoperations further comprise: indicating, by the primary storage system,updated tracks to mirror to the secondary storage system insynchronization information; determining, by the primary storage system,updated tracks to mirror for which track format information ismaintained; generating, by the primary storage system, a cache transferlist; and for each determined updated track of the determined updatedtracks, including, by the primary storage system, an entry in the cachetransfer list including a track identifier and the track formatinformation for the determined updated track, wherein the sending thetrack format information comprises sending the cache transfer list tothe secondary storage system, and wherein the sending the track tomirror comprises sending the updated tracks indicated in thesynchronization information to the secondary storage system.
 9. Thecomputer program product of claim 8, wherein the operations furthercomprise: storing, by the secondary storage system, received updatedtracks mirrored from the primary storage system in the secondary cache;generating by the secondary storage system, a cache control block foreach of the updated tracks mirrored to the secondary cache; determining,by the secondary storage system, whether the cache transfer listincludes track format information for the updated tracks; and including,by the secondary storage system, the track format information for thereceived updated tracks indicated in the cache transfer list in thecache control blocks for the received updated tracks in response to thedetermining that the cache transfer list includes the track formatinformation for the updated track.
 10. The computer program product ofclaim 8, wherein the sending the cache transfer list to the secondarystorage system comprises: sending, by the primary storage system, amessage to the secondary storage system including the cache transferlist before mirroring the updated tracks to the secondary storage systemfor which track format information is provided in the cache transferlist.
 11. A storage environment, including: a primary storage systemhaving a primary cache and a primary storage; a secondary storage systemhaving a secondary cache and a secondary storage; wherein the primarystorage system is to execute first computer program instructions toperform operations, the operations comprising: determining a track tomirror from the primary storage system to the secondary storage system;determining whether there is track format information for the track tomirror that the primary storage system maintains for caching the trackto mirror in the primary cache, wherein the track format informationindicates a format and layout of data in the track, indicated in trackmetadata for the track; sending the track format information to thesecondary storage system in response to determining there is the trackformat information; mirroring the track to mirror to the secondarystorage system; and wherein the secondary storage system is to executesecond computer program instructions to perform: storing, by thesecondary storage system, a mirrored track mirrored from the primarystorage system in the secondary cache; creating, by the secondarystorage system, a cache control block for the mirrored track;determining, by the secondary storage system, whether the primarystorage system provided track format information for the mirrored track;including, by the secondary storage system, the track format informationin the cache control block for the mirrored track in response todetermining that the primary storage system provided track formatinformation for the mirrored track; and using the track formatinformation for the track in the secondary cache when processing a reador write request to the mirrored track.
 12. The storage environment ofclaim 11, wherein the track format information comprises a track formatcode defined in a primary track format table at the primary storagesystem and a secondary track format table at the secondary storagesystem associating track format codes with track format metadata. 13.The storage environment of claim 11, wherein the track to mirrorcomprises an update track in the primary cache, wherein the primarystorage system is further to execute the first computer programinstructions to perform: determining whether the updated track modifiesthe format of the track; invalidating the track format informationmaintained for the track in the primary cache in response to determiningthat the updated track modifies the format of the track, wherein theinvalidated track format information for the track to mirror is not sentto the secondary storage system when mirroring the update track to thesecondary storage system; wherein the secondary storage system isfurther to execute the second computer program instructions to perform:rebuilding track metadata for the updated track modifying the format ofthe track; determining track format information from the track metadatarebuilt from the updated track; and including the determined trackformat information in a cache control block for the updated track in thesecondary cache. 14-15. (canceled)
 16. The storage environment of claim11, wherein the primary storage system is further to execute the firstcomputer program instructions to perform: indicating updated tracks tomirror to the secondary storage system in synchronization information;determining updated tracks to mirror for which track format informationis maintained; generating a cache transfer list; and for each determinedupdated track of the determined updated tracks, including an entry inthe cache transfer list including a track identifier and the trackformat information for the determined updated track, wherein the sendingthe track format information comprises sending the cache transfer listto the secondary storage system, and wherein the sending the track tomirror comprises sending the updated tracks indicated in thesynchronization information to the secondary storage system.
 17. Amethod for mirroring data from a primary storage system to a secondarystorage system comprising: determining, by the primary storage system, atrack to mirror from the primary storage system to the secondary storagesystem; determining, by the primary storage system, whether there istrack format information for the track to mirror that the primarystorage system maintains for caching the track to mirror in a primarycache, wherein the track format information indicates a format andlayout of data in the track, indicated in track metadata for the track;sending, by the primary storage system, the track format information tothe secondary storage system, in response to determining there is thetrack format information; mirroring, by the primary storage system, thetrack to mirror to the secondary storage system; storing, by thesecondary storage system, a mirrored track mirrored from the primarystorage system in a secondary cache; creating, by the secondary storagesystem, a cache control block for the mirrored track; determining, bythe secondary storage system, whether the primary storage systemprovided track format information for the mirrored track; including, bythe secondary storage system, the track format information in the cachecontrol block for the mirrored track in response to determining that theprimary storage system provided track format information for themirrored track; and using, by the secondary storage system, the trackformat information for the track in the secondary cache when processinga read or write request to the mirrored track.
 18. The method of claim17, wherein the track format information comprises a track format codedefined in a primary track format table at the primary storage systemand a secondary track format table at the secondary storage systemassociating track format codes with track format metadata.
 19. Themethod of claim 17, wherein the track to mirror comprises an updatetrack in the primary cache, further comprising: determining, by theprimary storage system, whether the updated track modifies the format ofthe track; invalidating, by the primary storage system, the track formatinformation maintained for the track in the primary cache in response todetermining that the updated track modifies the format of the track,wherein the invalidated track format information for the track to mirroris not sent to the secondary storage system when mirroring the updatetrack to the secondary storage system; rebuilding, by the secondarystorage system, track metadata for the updated track modifying theformat of the track; determining, by the secondary storage system, trackformat information from the track metadata rebuilt from the updatedtrack; and including, by the secondary storage system, the determinedtrack format information in a cache control block for the updated trackin the secondary cache. 20-21. (canceled)
 22. The method of claim 17,further comprising: indicating, by the primary storage system, updatedtracks to mirror to the secondary storage system in synchronizationinformation; determining by the primary storage system, updated tracksto mirror for which track format information is maintained; generating,by the primary storage system, a cache transfer list; and for eachdetermined updated track of the determined updated tracks, including, bythe primary storage system, an entry in the cache transfer listincluding a track identifier and the track format information for thedetermined updated track, wherein the sending the track formatinformation comprises sending the cache transfer list to the secondarystorage system, and wherein the sending the track to mirror comprisessending the updated tracks indicated in the synchronization informationto the secondary storage system.